Adaptive image interpolation for volume rendering

ABSTRACT

Sampling frequency of a ray casting for generating a projection image is varied in dependence of information derived from a 3D volume data during rendering. Furthermore, an interpolation is performed for skipped pixels for which no ray casting was performed in the projection image, based on this information.

The present invention relates to the field of digital imaging, inparticular where volume rendering is used for the visualization ofthree-dimensional data. In particular, the present invention relates toa method for rendering a three-dimensional volume dataset, to imageprocessing devices for rendering three-dimensional volume datasets andto computer programs for rendering three-dimensional volume datasets.

Volume rendering is a technique for visualizing sampled functions ofthree spatial dimensions by computing 2D projections of asemi-transparent volume. Volume rendering projects a volume dataset ontoa two-dimensional (2D) image plane or frame buffer. Volume rendering canbe used to view and analyze three-dimensional (3D) data from variousdisciplines, such as biomedicine, geophysics, computational fluiddynamics, finite element models and computerized chemistry. Volumerendering is also useful in the application of 3D graphics, such asvirtual reality (VR), computer aided design (CAD), computer games,computer graphics, special effects and the like. The variousapplications, however, may use a variety of terms, such as 3D datasets,3D images, volume images, stacks of 2D images and the like, to describevolume datasets.

Because of the increasing size of images, for example, in the medicaldomain, an interactive rendering speed at a high display qualitycontinues to be a challenging task, even though computationalperformance of computers is increasing.

In order to facilitate interactive visualization applications, imageswith a reduced resolution may be generated and scaled to the desiredhigh resolution afterwards. However, this results in a reduced imagingquality, since small details may appear blurred or distorted.

It is an object of the present invention to provide for an improvedrendering speed.

According to an exemplary embodiment of the present invention, the aboveobject may be solved by a method for rendering a three-dimensionalvolume dataset, wherein a sampling frequency in a projection image isvaried in dependence on information derived from the three-dimensionalvolume dataset. According to an aspect of the present invention, thisinformation is derived from the three-dimensional volume dataset duringthe rendering.

In other words, according to this exemplary embodiment of the presentinvention, a variation of the sampling in the projection image is basedon information derived from the three-dimensional volume datasets duringrendering.

Advantageously, according to this exemplary embodiment of the presentinvention, an interactive visualization at a high speed is facilitated,while maintaining a high image quality. Compared, for example, with animage up-sampling, this exemplary embodiment of the present inventionmay lead to an improved image quality with no or only negligibleperformance loss. Compared to a rendering at full resolution, thisexemplary embodiment of the present invention may facilitate speed-upsby a factor of 3 (for up-sampling by a factor of 2 in each dimension;larger speed-ups for higher up-sampling factors). However, since thespeed-up may vary with the respective hardware, different speed-upfactors may be achieved.

According to other exemplary embodiments of the present invention, therendering includes a ray casting, which involves the computation ordetermination of depth values and light reflection angles. According toan aspect of the present invention, the sampling frequency in theprojection image is varied to at least one of a depth value, surfacenormal, viewing direction and light vector.

According to these exemplary embodiments of the present invention, a raycasting is used to generate, i.e. the projection image with reducedresolution. In dependence of the depth values and light reflectionangles determined during the ray casting, areas of the projection imageshowing details are identified and the sampling frequency is increasedfor these areas, i.e. additional rays are cast to increase theresolution in these areas.

According to another exemplary embodiment of the present invention, thevariation of the sampling frequency is increased in case a differencebetween angles between viewing vectors and a surface normal of a pair ofpixels in the projection image exceeds a first predetermined threshold,a difference of depth values of a pair of pixels in the projection imageexceeds a second predetermined threshold or a depth value of a pixel inthe projection image exceeds a predetermined threshold. According tothis exemplary embodiment of the present invention, a very fast andefficient method is provided for identifying areas of the projectionimage, where the sampling frequency has to be increased and additionalrays are to be cast to increase the resolution in areas where a lowsampling frequency is insufficient or areas where the sampling frequencymay even be reduced.

According to other exemplary embodiments of the present invention, a raycasting is performed only for each second, third, fourth etc. pixel ofthe projection image. According to an aspect of these exemplaryembodiments of the present invention, pixels in the projection image forwhich no ray casting has been performed, are interpolated on the basisof the information derived from the three-dimensional dataset during therendering, such as, for example, depth values and light reflectionangles of neighboring pixels for which a ray casting has been performed.In other words, according to these exemplary embodiments of the presentinvention, in areas where no additional ray casting has been performed,such that there are pixels in the projection image for which no imageinformation has been obtained, depth values and light reflection anglesof surrounding pixels are interpolated and a pixel shading is applied togenerate the missing pixel, i.e. to determine the missing pixelinformation.

Advantageously, due to the interpolation based on the informationderived during the rendering, in particular the ray casting, an improvedfinal image may be provided, without blurs or distortions.

According to another exemplary embodiment of the present invention, amethod is provided, wherein a projection image generated during therendering has a first number of pixels. According to an aspect of thepresent invention, a ray casting is performed for a second number ofpixels, wherein the second number is smaller than the first number. Toallow for an improved image quality, while reducing the computationalefforts necessary to display the three spatial dimensions in the final212 projection, pixels, for which no ray casting has been performed, areinterpolated on the basis of information derived from thethree-dimensional volume dataset during ray casting. According to theexemplary embodiment of the present invention, this informationcorresponds to at least one of depth values and light reflection angles.

According to this exemplary embodiment of the present invention, aninteractive visualization at high speed is facilitated. Furthermore,images may be computed in a very short time, which have an improvedquality compared with, for example, image up-sampling. Compared torendering at full resolution, the above exemplary embodiment of thepresent invention may allow for a significant speed-up by factors of atleast 3 or more, while maintaining a very good image quality.

According to other exemplary embodiments of the present invention, imageprocessing devices are provided, each including a memory and aprocessor. According to an aspect of the present invention, these imageprocessing devices are adapted to perform the methods according to thepresent invention. According to these exemplary embodiments of thepresent invention, image processing devices are provided, which allowfor a very fast visualizing of sampled functions of three spatialdimensions in 2D projections. Furthermore, due to the reduced amount ofcalculating steps required, a processor with a reduced calculation powermay be used, which allows to significantly reduce the costs of the imageprocessing devices according to the present invention.

According to other exemplary embodiments of the present invention,computer programs are provided, causing a processor to perform anoperation when the computer program is executed on the processors,corresponding to the methods according to the present invention. Thesecomputer programs may be written in any suitable programming language,for example, C++ and may, for example, be stored on a computer readablemedium, such as a CD-ROM. Also, these computer programs may be availablefrom a network, such as the WorldWideWeb, from which they may bedownloaded to image processing units or processors, or any suitablecomputer.

It may be seen as the gist of an exemplary embodiment of the presentinvention that the sampling of the projection image is varied independence of information derived from the three-dimensional volumedataset during rendering. Furthermore, according to an aspect of thepresent invention, skipped pixels in the projection image areinterpolated on the basis of this information derived during rendering.This information may, for example, correspond to at least depth valuesor light reflection angles.

These and other aspects of the present invention will become apparentfrom an elucidated with reference to the embodiments describedhereinafter.

Exemplary embodiments of the present invention will be described in thefollowing, with reference to the following drawings:

FIG. 1 shows a schematic representation of an image processing deviceaccording to an exemplary embodiment of the present invention, adaptedto execute an exemplary embodiment of a method according to the presentinvention.

FIG. 2 shows a simplified representation for further explaining anexemplary embodiment of the present invention.

FIG. 3 shows another simplified representation for further explaining anexemplary embodiment of the present invention.

FIG. 4 shows areas where a sampling has been performed with an increasedsampling frequency in the Figure of FIG. 5 in accordance with anexemplary embodiment of the present invention.

FIG. 5 shows an example of an endoscopic image determined in accordancewith an exemplary embodiment of the present invention.

FIG. 1 shows an exemplary embodiment of an image processing deviceaccording to the present invention. The image processing device shown inFIG. 1 includes an image processing and control processor 1 with amemory 9, in which a three-dimensional volume dataset and a projectionimage generated during rendering may be stored. The image processing andcontrol processor (CPU) 1 may be coupled, via a bus system 3, to animaging device (not shown in FIG. 1), for example, an MR apparatus, a US(ultrasonic) apparatus, or a CT apparatus. An image generated by theimage processing and control processor 1 may be displayed to an operatoron a monitor 7, connected to the image processing and control processor1. The operator may access the image processing and control processor 1via a keyboard 5 or other input means which are not shown in FIG. 1,such as a mouse or a trackball.

In the following, the present invention will be described with referenceto the example of a virtual endoscopy or virtual colonoscopy wheremovies showing a fly through the colon are generated to provide an easyway of spotting anomalies of the colon wall. However, it has to be notedthat the present invention is not limited to this example or to anymedical applications, but can be applied and used in any type ofapplication dealing with the visualization of objects or modelsrepresented as sample data in three or more dimensions.

FIG. 2 shows a simplified representation depicting how a 2D image isgenerated from volume data.

Reference numeral 2 in FIG. 2 designates the three-dimensional volumedataset, i.e. the volume data. As already mentioned above, the volumedata may, for example, be acquired by means of an ultrasonic apparatus,CT apparatus or MRI apparatus. However, as also noted above thisthree-dimensional volume dataset may also be generated by means of acomputer and represent computer graphics. In a first step, from thethree-dimensional volume dataset 2, intermediate data 4 for representinga projection image is generated. This may be done by ray casting, wherea ray casting is performed for each pixel S_(0,0) to S_(4,4) of theprojection image in order to calculate a distance from a view point tothe corresponding surface point and an angle between a viewing vectorand a surface normal at that point.

Such a method is, for example, described in further detail in M. Levoy“Display of Surfaces from Volume Data”, IEEE Computer Graphics andApplication, Vol. 8, No. 5, pp. 29-37, 1988, which is herebyincorporated by reference.

As may be taken from FIG. 2, each pixel S_(0,0) to S_(4,4) may containinformation with respect to at least one of depth values, lightreflection angles and the surface normal at that point. Also,information relating to viewing direction and light vector may beincluded.

Then, in a subsequent step, this intermediate data 4, i.e. theprojection image, may be transformed into gray values or color values 8,based on a shading model, to give a realistic impression of the anatomy.A suitable shading model is, for example, known from Phong, Bui-Tuong,“Illumination for Computer-Generated Pictures”, Communications of theACM, Vol. 18, No. 3, pp. 311-317, 1975, which is hereby incorporated byreference.

From a complexity point of view, the ray casting step between thethree-dimensional volume dataset 2 and the intermediate data 4, may bethe most critical step with respect to the speed, as it involves theprocessing of large three-dimensional datasets, including tri-linear ortri-cubic interpolation. Furthermore, this step may require costly andirregular memory access. A CPU time required for shading can beneglected in comparison to the CPU time required for ray casting betweenthe three-dimensional volume dataset 2 and the intermediate data 4, asthe total amount of data to be processed is one order of magnitudelower, namely two-dimensional instead of three-dimensional.

FIG. 3 shows a simplified representation for further explaining a methodof operating the image processing device of FIG. 1 in accordance with anexemplary embodiment of the present invention. The method described withreference to FIG. 3 may also be implemented as a computer programwritten in a suitable programming language, such as, for example, C++.Such a computer program may be stored on a computer readable medium,such as a CD-ROM. Or the computer program may be presented via anetwork, such as the WorldWideWeb from which it can be downloaded into amemory of a processor, suitable computer or suitable image processingdevice.

As may be taken from FIG. 3, after the acquisition of thethree-dimensional volume dataset 2, an adaptive ray casting is performedto obtain the intermediate data 4, namely the projection image, such aray casting is, for example, described in M. Levoy “Display of Surfacesfrom Volume Data”, IEEE Computer Graphics and Application, Vol. 8, No.5, pp. 29-37, 1988, which is hereby incorporated by reference. However,as may be taken from FIG. 3, the ray casting is first performed only forevery n^(th) pixel of the projection image, with n ε [2; 3; 4; . . . ∞[.In the example of FIG. 3, the ray casting is initially only performedfor every second pixel of the projection image in both dimensions, i.e.x and y dimensions of the projection image and thus only 25% of thetotal number of pixel of the projection image. Then, in a subsequentstep, according to the present invention, areas of the projection imageare determined, where this undersampling would cause a significant lossof image quality. In other words, in this step, areas in the projectionimage are identified, where a reduced sampling frequency may be acceptedand does not cause a deterioration of the image quality and areas wheresuch a reduced sampling or undersampling would cause a significant lossof image quality and thus where an increase of the sampling frequency isrequired. Areas, where the undersampling would cause a significant lossof image quality are, for example

-   -   areas with high variations of the surface orientation    -   areas including edges, for example, at wrinkles of the surface        and    -   distant areas in the final image, where the surface structures        appear small as a result of the perspective projection.

These areas may be identified as follows:

For a given image pixel p of the pixels of the projection images S_(0,0)to S_(4,4), α_(p) is the angle between the viewing vector and thesurface normal, which is also referred to as image gradient. Thequantity z_(p) denotes the depth value of the pixel. Further, Δz_(p) andΔα_(p) are the maximum differences of respective values for neighboringpixels. These values α_(p), z_(p), Δα_(p) and Δ_(p) can be determinedfrom the results of the ray casting step.

In order to ensure that a subsequent interpolation of the pixel p doesnot smear out image qualities, i.e. doe not cause blurring ordistortions in the final image, the following criteria may be formulatedon the basis of which the above areas, where an undersampling wouldcause a significant loss of image quality may be identified. Also, thesecriteria ensure that a subsequent interpolation allows a sufficientlygood interpolation of the skipped pixel p, for which no ray casting wasperformed.

According to the first criterion, the sampling frequency is increased incase a difference Δα_(p) of the angles α_(p) between the viewing factorand the surface normals of a pair of pixels exceed a given limitΔα_(max). In other words, the sampling frequency is increased and anadditional ray casting is performed for a pixel between a pair of pixelsor surrounded by at least two pixels, for which this criterion isfulfilled, i.e. where the difference between the angles between theviewing factor and the surface normal for these two pixels exceeds athreshold value. This criterion can be formulated as Δα_(p)>Δα_(max).

The second criterion is based on a difference of depth values of a pairof pixels. According to this criterion, in case the difference of thedepth values of a pair of pixels exceeds a given limit or threshold, thesampling frequency for pixels neighboring or in the vicinity of thispair of pixels for which this criterion is fulfilled, is increased andan additional ray casting is performed. This second criterion may beformulated asΔz_(p)>Δz_(max).

The third criterion is based on the depth value. According to thiscriterion, the sampling frequency is increased, in case a depth value islarger than a given limit or threshold. In contrast to the first andsecond criteria, this criterion is formulated for a single pixel and notfor pairs of pixels. Mathematically, this criterion can be described bythe following equation: z_(p)>z_(max).

It has to be noted that, instead of relating to pairs of pixels, theabove method can easily be extended to 3, 4 or a plurality of pixels.

It has to be noted that the present invention may also be applied toparallel projection. In such a case, the third criterion is not used.

As mentioned above, the initial ray casting in FIG. 3 was only performedfor every second pixel in both dimensions. For the remaining pixels, asindicated by the bar in the projection image, no ray casting wasperformed. Then, after the ray casting with reduced sampling frequency,the above three criteria are checked for the projection image. As may betaken from FIG. 3, for the pixel S_(3,0) one of the above criteria ismet. In accordance with the present invention, due to the fact that atleast one of the above three criteria was met, the sampling frequency isincreased and an additional ray casting, as indicated by the broken line6, is performed for this pixel S_(3,0). In other words, for S_(3,0), thedepth value z_(3,0) and the angle α_(3,0) between the viewing vector andthe surface normal, is determined exactly by ray casting.

Then, in a subsequent step, the pixels for which no ray casting wasperformed, indicated in FIG. 3 by bars in the projection image orimmediate data 4, an interpolation is performed. During thisinterpolation, according to an aspect of the present invention, thesepixels S′_(0,1) S′_(0,3) S′_(1,0) S′_(1,1) S′_(1,2) S′_(1,3) S′_(1,4)S′_(2,1) S′_(2,3) S′_(3,1) S′_(2,1) S′_(3,2) S′_(3,3) S′_(3,4) S′_(4,3)and S′_(4,3) are interpolated from information determined during the raycasting. In other words, these pixel values are determined on the basisof at least one of depth values and light reflection angles from thesurrounding pixels, for which a ray casting was performed.

After the remaining z and α values for the pixels for which no raycasting was performed have been interpolated, in a subsequent step, ashading is performed at full resolution, in order to generate the finalimage 8 displayed on the monitor 7 to an operator. A suitable shadingmethod is described, for example, Phong, Bui-Tuong, “Illumination forComputer-Generated Pictures”, Communications of the ACM, Vol. 18, No. 3,pp. 311-317, 1975, which is hereby incorporated by reference.

According to the above described method, the sampling in the projectionimage is varied on the basis of information derived from thethree-dimensional volume dataset during the rendering, i.e. the raycasting. Furthermore, according to the present invention, aninterpolation of skipped pixels is performed to determine imageinformation for these skipped pixels for which no ray casting wasperformed. In other words, ray casting is used to generate, forinstance, a surface rendering with reduced solution. In dependence ofdepth values and light reflection angles from the ray casting step,areas showing details are identified and additional rays are cast toincrease the resolution. In other areas, depth values and lightreflection angles are interpolated and pixel shading is applied togenerate missing pixels.

Compared with an image up-sampling, the above described exemplaryembodiment of the present invention allows for an improved image qualitywith no or only negligible performance loss or speed-loss. Compared, forexample, to a rendering at full resolution, the above described methodmay facilitate a speed-up by at least a factor of 3, for example, for anup-sampling by a factor of 2 in each dimension. A larger up-samplingfactor naturally causes a higher speed-up. However, also smallerspeed-up factors may be achieved.

In the method described with reference to FIG. 3, firstly, all exactpixels of the projection image are determined, and then the criteria arechecked, and further pixels are determined by an additional ray-casting.Finally, the interpolation is performed.

According to an aspect of the present invention, these steps are notperformed separately one after another, but are performed in combinationtogether for selected (partial) areas/regions of the projection image.

As mentioned above, the sampling frequency may be varied on the basis ofat least one of a depth value, surface normal, viewing direction andlight vector. According to the preferred embodiment of the presentinvention, the sampling frequency is varied on the basis of surfacegradient vectors, i.e. angles between surface normals and the viewingdirection and the depth values.

FIG. 5 shows an example of an endoscopic image, for which the abovedescribed method was applied. FIG. 4 shows the density of exactlycomputed rays, where the darker regions of FIG. 4 show areas where noincreased sampling frequency was used and the lighter areas show areaswhere an increased sampling frequency has been used. The image size is300² pixels and, according to the above explained exemplary embodimentof the present invention, the initial ray casting was performed suchthat every second pixel in each dimension was covered by the raycasting. Thus, a ray casting was performed for 25% of all pixels.Approximately 12% of the remaining pixels did satisfy one of the abovethree criteria. Thus for this 12% of pixels, an additional ray castingwas performed, i.e. the sampling frequency was increased for thesepixels. Thus, in comparison to a rendering at full resolution, where raycasting is performed for each pixel, in FIGS. 4 and 5, 63% of theexpensive ray casting operation has been eliminated, thus allowing for asignificant speed-up of the rendering operation. In the case of FIGS. 4and 5, this led to a speed-up of a factor of about 3.

As described above, the present invention uses two mechanisms, namely,an adaptive ray casting with a varying sampling frequency, and a datainterpolation in the intermediate data 4, i.e. the projection image.According to an aspect of the present invention, both mechanisms neednot be used in conjunction. The adaptive sampling may be used incombination with other rendering techniques, such as, for example, themaximum intensity projection (MIP) or the direct volume rendering.

The interpolation of intermediate data may also be used as analternative to an image-domain reformatting and may provide for a betterimage quality, because the intermediate image representation containsinformation, which is lost in the final image.

Also, both mechanisms according to the present invention may be appliedto the compression of two-dimensional images and movies generated byrenderings from 3D datasets.

1. A method for rendering a three dimensional volume data set, themethod comprising the acts of: varying, via a processor, a samplingfrequency in a projection image in dependence of information derivedfrom the three dimensional volume data set; and rendering, via aprocessor, a resulting image to an output device; wherein theinformation is derived from the three dimensional volume data set duringthe rendering; wherein the sampling frequency is increased when a depthvalue of a pixel in the projection image exceeds a predeterminedthreshold; and wherein the sampling frequency is increased when adifference between angles between viewing vectors and a surface normalof a pair of pixels in the projection image exceeds a furtherpredetermined threshold.
 2. A method for rendering a three dimensionalvolume data set, wherein a projection image generated during therendering has a first number of pixels, the method comprising the actsof: performing, via a processor, a ray casting for a second number ofpixels, wherein the second number is smaller than the first number;interpolating, via a processor, pixels of the first number of pixel ofthe projection image for which no ray casting was performed frominformation determined during the ray casting; and rendering a resultingimage to an output device; wherein a sampling frequency in theprojection image is increased when a depth value of a pixel in theprojection image exceeds a predetermined threshold; and wherein thesampling frequency is increased when a difference between angles betweenviewing vectors and a surface normal of a pair of pixels in theprojection image exceeds a further predetermined threshold.
 3. An imageprocessing device for rendering a three dimensional volume data set, theimage processing device comprising: a memory for storing the threedimensional volume data set and a projection image generated duringrendering; and an image processor adapted to vary a sampling frequencyin the projection image in dependence of information derived from thethree dimensional volume data set; wherein the information is derivedfrom the three dimensional volume data set during the rendering; whereinthe sampling frequency is increased when a depth value of a pixel in theprojection image exceeds a predetermined threshold; and wherein thesampling frequency is increased when a difference between angles betweenviewing vectors and a surface normal of a pair of pixels in theprojection image exceeds a further predetermined threshold.
 4. An imageprocessing device for rendering a three dimensional volume data set,wherein a projection image generated during the rendering has a firstnumber of pixels, the image processing device comprising: a memory forstoring the three dimensional volume data set and the projection imagegenerated during rendering; and an image processor adapted to performthe following operation: performing a ray casting for a second number ofpixels, wherein the second number is smaller than the first number; andinterpolating pixels of the first number of pixels of the projectionimage for which no ray casting was performed from information determinedduring the ray casting; wherein a sampling frequency in the projectionimage is increased when a depth value of a pixel in the projection imageexceeds a predetermined threshold; and wherein the sampling frequency isincreased when a difference between angles between viewing vectors and asurface normal of a pair of pixels in the projection image exceeds afurther predetermined threshold.
 5. A non-transitory computer-readablemedium comprising a computer program for rendering a three dimensionalvolume data set, the computer program being adapted to cause a processorto perform the following operation when the computer program is executedon the processor: varying a sampling frequency in a projection image independence of information derived from the three dimensional volume dataset; wherein the information is derived from the three dimensionalvolume data set during the rendering; wherein the sampling frequency isincreased when a depth value of a pixel in the projection image exceedsa predetermined threshold; and wherein the sampling frequency isincreased when a difference between angles between viewing vectors and asurface normal of a pair of pixels in the projection image exceeds afurther predetermined threshold.
 6. A non-transitory computer-readablemedium comprising a computer program for rendering a three dimensionalvolume data set, wherein a projection image generated during therendering has a first number of pixels, the computer program beingadapted to cause a processor to perform the following operation when thecomputer program is executed on the processor: performing a ray castingfor a second number of pixels, wherein the second number is smaller thanthe first number; and interpolating pixels of the first number of pixelsof the projection image for which no ray casting was performed frominformation determined during the ray casting; wherein a samplingfrequency in the projection image is increased when a depth value of apixel in the projection image exceeds a predetermined threshold; andwherein the sampling frequency is increased when a difference betweenangles between viewing vectors and a surface normal of a pair of pixelsin the projection image exceeds a further predetermined threshold.